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Abstract 


The  Department  of  Defense  (DOD)  has  faced  significant  challenges 
managing  the  total  ownership  cost  (TOC),  schedule,  and  technical  performance  of 
software-intensive  systems.  These  challenges  will  continue  to  grow  as  proposed, 
and  future  systems  will  depend  on  software  for  an  ever-increasing  portion  of  system 
functionality,  requiring  the  development  of  larger  and  more  complex  software 
applications.  In  addition,  the  development  of  the  envisioned  tactical  and  strategic 
net-centric  warfighting  systems  will  require  unprecedented  software  development 
efforts. 

This  research  is  a  continuation  and  consolidation  of  previous  research 
projects  conducted  for  the  US  Navy  Open  Architecture  Task  Force.  That  previous 
research  is  identified  and  cited  where  appropriate. 

The  purpose  of  this  research  is  to  analyze  numerous  tools,  techniques,  and 
processes  combined  in  a  unique  way  to  provide  more  predictability  and  control  to  the 
software  development  within  the  restrictive  DOD  Acquisition  Management  System. 
The  tools  and  analyses  include  the  Software  Engineering  Institute’s  Quality 
Attribution  Workshop  (QAW),  the  MUIRS  (maintainability,  upgradability, 
interoperability,  reliability,  and  safety/security)  analysis  methodology,  SEI’s 
Architectural  Tradeoff  Analysis  Methodologysm  (ATAMsm),  Logistics  Supportability 
Analysis  (LSA),  and  the  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA).  In 
addition,  the  concept  of  software  Management  Readiness  Levels  (MgtRLs)  are 
introduced  as  a  more  useful  risk  reduction  technique  as  compared  to  the  software 
Technology  Readiness  Levels  (TRLs)  currently  used. 

This  research  demonstrates  how  the  combined  tools,  analyses,  and 
processes  address  the  most  common  DOD  software-intensive  system 
developmental  issues  in  a  unique  and  holistic  way.  Although  each  tool,  analysis 
technique,  and  process  has  individual  utility  and  is  value-added,  this  research 
demonstrates  how  the  combined  use  produces  a  synergistic  solution  to  the  software 
component  development  control  and  produces  significantly  more  predictability  in  the 
program  management  realm. 

The  research  conclusions  and  recommendations  are  designed  to  provide 
current  and  future  DOD  Program  Managers  with  the  combined  tools,  analyses,  and 
processes  within  a  conceptual  implementation  scheme  that  will  provide  more  control 
and  predictability  to  software-intensive  systems  development.  Due  to  the  TOC  and 
architectural  design  focus,  system  sustainability  costs  are  thoroughly  addressed  and 
actively  managed. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Keywords:  Software-intensive  system  acquisition,  system  acquisition  control 
and  predictability,  software  system  sustainability,  software  system  management, 
Quality  Attribute  Workshop  (QAW),  Architecture  Trade-off  Analysis  Methodology 
(ATAM),  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA),  MUIRS,  software 
architecture,  system  software  design,  metrics,  Joint  Capabilities  Integration  and 
Development  System  (JCIDS),  Systems  Engineering  Process  (SEP),  DOD 
Acquisition  System. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Acknowledgments 


The  author  would  like  to  thank  Rear  Admiral  (Ret.)  James  Greene  for  his 
remarkable  contribution  to  the  DOD  acquisition  community  as  the  Acquisition  Chair 
for  the  Naval  Postgraduate  School.  His  efforts  have  enabled  these  research  efforts 
that  address  current  and  future  problems,  and  vastly  improve  the  education  product 
that  NPS  delivers. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


About  the  Author 


Brad  R.  Naegle,  Lieutenant  Colonel,  U.S.  Army  (Ret.),  is  a  Senior  Lecturer 
and  Academic  Associate  for  Program  Management  Curricula  at  the  Naval 
Postgraduate  School,  Monterey,  CA.  In  addition  to  acquisition  course  development 
and  delivery,  he  is  the  Academic  Associate  for  the  Master  of  Science  in  Program 
Management  curriculum,  which  is  a  Distance  Learning  Master’s  program,  serving 
students  around  the  world.  While  on  active  duty,  LTC  (Ret.)  Naegle  was  assigned  as 
the  Product  Manager  for  the  2V2-ton  Extended  Service  Program  (ESP)  and  USMC 
Medium  Tactical  Vehicle  Replacement  (MTVR)  from  1994  to  1996  and  served  as  the 
Deputy  Project  Manager  for  Light  T actical  Vehicles  from  1 996  to  1 997.  He  was  the 
7th  Infantry  Division  (Light)  Division  Materiel  Officer  from  1990  to  1993  and  the  34th 
Support  Group  Director  of  Security,  Plans,  and  Operations  from  1986  to  1987.  Prior 
to  that,  LTC  (Ret.)  Naegle  held  positions  in  test  and  evaluations  and  logistics  fields. 
He  earned  a  Master  of  Science  degree  in  systems  acquisition  management  (with 
Distinction)  from  the  Naval  Postgraduate  School  and  an  undergraduate  degree  in 
economics  from  Weber  State  University.  He  is  a  graduate  of  the  Command  and 
General  Staff  College,  Combined  Arms  and  Services  Staff  School,  and  Ordnance 
Corps  Advanced  and  Basic  Courses. 

Brad  Naegle 
Senior  Lecturer 

Graduate  School  of  Business  &  Public  Policy 
Naval  Postgraduate  School 
Monterey,  CA  93943 
Tel:  (831)  656-3620 
E-mail:  bnaeqle@nps.edu 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


NPS-AM-14-194 


ACQUISITION  RESEARCH  PROGRAM 
SPONSORED  REPORT  SERIES 


Gaining  Control  and  Predictability  of  Software-Intensive  Systems 

Development  and  Sustainment 

4  February  2015 

Brad  Naegle,  Senior  Lecturer 

Graduate  School  of  Business  &  Public  Policy 

Naval  Postgraduate  School 


Disclaimer:  The  views  represented  in  this  report  are  those  of  the  author  and  do  not  reflect  the  official  policy 
position  of  the  Navy,  the  Department  of  Defense,  or  the  federal  government. 


mtSTANTlAPERSCIfNTT/Ui 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  VII  - 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Table  of  Contents 


Executive  Summary . xiii 

Introduction . 1 

The  DOD  Software-Intensive  System  Development  Problem  and  Research 
Technique . 3 

Problem . 3 

Primary  Research  Question . 3 

Secondary  Research  Questions . 4 

DOD  Acquisition  Environment . 4 

Requirements  Generation . 5 

The  Defense  Acquisition  System . 6 

Performance  Specifications  and  the  Work  Breakdown  Structure . 7 

Technology  Readiness  Assessment  and  Risk  Management . 8 

Findings  Summary . 10 

DOD  Acquisition  Environment  Analysis . 10 

Software  Engineering  Environment . 1 1 

Software  Engineering . 11 

Comparison  to  Mature  Engineering . 12 

Findings  Summary . 13 

Software  Engineering  Environment  Analysis . 13 

DOD  Acquisition  Environment:  Impact  on  Software  Development  and  Quality 
Attributes . 14 

DOD  Requirements  Generation  Process . 14 

Systems  Engineering  Process . 14 

Work  Breakdown  Structure . 15 

Software  Engineering  Maturity  Impact  on  Requirements  Generation . 16 

System  Operational  Context . 17 

Impact  on  Software  and  Quality  Attributes  Analysis . 19 

Software-Intensive  System  Architecture  Development  Analysis . 20 

Conclusions . 21 


mtSTANTlAPERSCIfNTT/Ui 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  ix  - 


Recommendations . 23 

General . 23 

Tools,  Techniques,  and  Processes . 23 

Quality  Attribute  Workshop . 24 

Maintainability,  Upgradability,  Interoperability/Interfaces,  Reliability,  and 
Safety/Security  Analytic  Technique . 24 

Architectural  Tradeoff  Analysis  Methodologysm . 25 

Failure  Modes  and  Effects  Criticality  Analysis . 28 

Integrating  the  Recommended  Tools,  Techniques,  and  Processes  into  the 
Defense  Acquisition  System . 29 

Program  Management  Risk  Reduction . 31 

References . 34 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


List  of  Figures 


Figure  1 .  Systems  Engineering  Process . 3 

Figure  2.  DOD  Decision  Support  Systems . 4 

Figure  3.  Defense  Acquisition  Management  System . 7 

Figure  4.  Quality  Attribution  Workshop  and  Architectural  T radeoff  Analysis 

Methodology  Integration  into  Software  Life-Cycle  Management ....  27 

Figure  5.  Quality  Attribution  Workshop  and  Architectural  T radeoff  Analysis 

Methodology  Integration  into  Software  Lifecycle  Management . 30 


p  ^tSUNTIAI’ERSCIfNTlAM 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  xi  - 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Executive  Summary 


The  Department  of  Defense  (DOD)  has  faced  significant  challenges 
managing  the  total  ownership  cost  (TOC),  schedule,  and  technical  performance  of 
software  intensive  systems.  These  challenges  will  continue  to  grow  as  proposed, 
and  future  systems  will  depend  on  software  for  an  ever-increasing  portion  of  system 
functionality,  requiring  the  development  of  larger  and  more  complex  software 
applications.  In  addition,  the  development  of  the  envisioned  tactical  and  strategic 
net-centric  warfighting  systems  will  require  unprecedented  software  development 
efforts. 

This  research  is  a  continuation  and  consolidation  of  previous  research 
projects  conducted  for  the  US  Navy  Open  Architecture  Task  Force.  That  previous 
research  is  identified  and  cited  where  appropriate. 

The  purpose  of  this  research  is  to  analyze  numerous  tools,  techniques,  and 
processes  combined  in  a  unique  way  to  provide  more  predictability  and  control  to  the 
software  development  within  the  restrictive  DOD  Acquisition  Management  System. 
The  tools  and  analyses  include  the  Software  Engineering  Institute’s  Quality 
Attribution  Workshop  (QAW),  the  MUIRS  (maintainability,  upgradability, 
interoperability,  reliability,  and  safety/security)  analysis  methodology,  SEI’s 
Architectural  Tradeoff  Analysis  Methodologysm  (ATAMsm),  Logistics  Supportability 
Analysis  (LSA),  and  the  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA).  In 
addition,  the  concept  of  software  Management  Readiness  Levels  (MgtRLs)  are 
introduced  as  a  more  useful  risk  reduction  technique  as  compared  to  the  software 
Technology  Readiness  Levels  (TRLs)  currently  used. 

This  research  demonstrates  how  the  combined  tools,  analyses,  and 
processes  integrate  with  the  Defense  Acquisition  System  (DAS)  and  address  the 
most  common  DOD  software-intensive  system  developmental  issues  in  a  unique 
and  holistic  way.  Although  each  tool,  analysis  technique,  and  process  has  individual 
utility  and  is  value-added,  this  research  demonstrates  how  the  combined  use 
produces  a  synergistic  solution  to  the  software  component  development  control  and 
produces  significantly  more  predictability  in  the  program  management  realm. 

The  research  conclusions  and  recommendations  are  designed  to  provide 
current  and  future  DOD  Program  Managers  with  the  combined  tools,  analyses,  and 
processes  within  a  conceptual  implementation  scheme  that  will  provide  more  control 
and  predictability  to  software-intensive  systems  development.  Due  to  the  TOC  and 
architectural  design  focus,  system  sustainability  costs  are  thoroughly  addressed  and 
actively  managed. 
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Introduction 


From  remotely  piloted  aircraft  and  smart  bombs  to  autonomous 
vehicles  and  advanced  fighter  jets,  software  is  crucial  to  the  success  of 
today’s  weapon  systems.  Focusing  solely  on  developing  and 
maintaining  military  hardware  is  no  longer  an  option.  With  shrinking 
defense  budgets  and  increasingly  complex  systems,  the  defense 
industry  and  services  must  fight  to  deliver  on  this  ambitious  objective, 
the  military  must  drastically  transform  its  approach  to  software.  New 
organizational  structures,  operating  models,  and  tools  will  be  essential 
to  modernizing  and  sustaining  the  U.S.  weapon  systems.  (Hagen,  Hurt, 
&  Sorenson,  2013,  p.  31) 


Although  the  Department  of  Defense  (DOD)  has  developed  some  very 
successful  software-intensive  systems,  such  as  the  Aegis,  Tomahawk  Missile,  and 
F/A-18  Hornet,  we  continue  to  struggle  with  successfully  developing  like  systems. 
The  software  development  in  the  F-35  Joint  Strike  Fighter  (JSF)  continues  to  be 
problematic.  The  Government  Accountability  Office  (GAO;  2012)  stated  that 


JSF  software  development  is  one  of  the  largest  and  most  complex 
projects  in  DoD  history,  providing  essential  capability,  but  software  has 
grown  in  size  and  complexity,  and  is  taking  longer  to  complete  than 
expected.  Developing,  testing,  and  integrating  software,  mission 
systems,  and  logistics  systems  are  critical  for  demonstrating  the 
operational  effectiveness  and  suitability  of  a  fully  integrated,  capable 
aircraft  and  pose  significant  technical  risks  moving  forward,  (p.  7) 


The  report  goes  on  to  state,  “This  program  [JSF]  has  modified  the  software 
development  and  integration  schedule  several  times,  in  each  instance  lengthening 
the  time  needed  to  complete  work”  (GAO,  2012,  p.  1 1 )  The  results  of  the  software 
development  problems  have  contributed  to  a  two-year  delay  and  increased  costs  of 
about  one  billion  dollars. 

When  software-intensive  systems  encounter  developmental  problems,  it  is 
easy  to  see  the  symptoms:  schedule  overruns,  acquisition  cost  overruns,  systems 
delivered  with  less  capability  than  desired,  and  unaffordable  software  sustainment 
costs.  The  actual  causes  of  the  visible  symptoms  are  often  much  more  difficult  to 
determine. 

Cost  and  schedule  overruns  in  software  development  are  often  the  result  of 
poor  initial  software  size  estimates  and  unforeseen  software  redesign.  In  the  case  of 
the  JSF, 


The  lines  of  code  necessary  for  the  JSF’s  capabilities  have  now  grown 
to  over  24  million — 9.5  million  on  board  the  aircraft.  By  comparison, 
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JSF  has  about  3  times  more  on-board  software  lines  of  code  than  the 
F-22A  Raptor  and  6  times  more  than  the  F/A-18  E/F  Super  Hornet. 

This  has  added  work  and  increased  the  overall  complexity  of  the  effort. 
The  software  on-board  the  aircraft  and  needed  for  operations  has 
grown  37  percent  since  the  critical  design  review  in  2005  ...  almost  half 
of  the  on-board  software  has  yet  to  complete  integration  and  test  - 
typically  the  most  challenging  phase  of  software  development.  (GAO, 

2012,  p.  11) 

The  report  goes  on  to  state  that  typical  software  size  growth  in  DoD  systems 
development  ranges  from  30%  to  100%. 

JSF  design  changes  were  originally  supposed  to  taper  off  and  be  completed 
by  January  2014.  Actual  design  changes  through  September  201 1  failed  to  taper  off 
and  continue  at  a  significantly  high  rate.  The  projections  in  the  GAO  (2012)  report 
indicated  that  the  revised  design  change  projections  will  continue,  and  actually  grow 
in  number,  until  January  2019  (p.  16).  Given  this  level  of  redesign,  the  software  and 
system  complexity  growth  are  likely  to  continue. 

Software  engineers  typically  spend  50%  or  more  of  their  total  software 
development  time  designing  software  architecture,  and  that  architecture  may  provide 
up  to  80%  of  a  modern  weapon  system’s  functionality.  Increasingly,  these  systems 
operate  within  a  network  or  other  system-of-systems’  architecture.  Obviously,  the 
requirements  driving  that  architectural  design  effort  are  critical  for  achieving  the 
warfighter  capability  sought.  Managing  the  architectural  design  process  (including 
tracing  requirement  to  functions),  insight  into  the  design  process,  and  control  of  the 
design  effort  are  equally  critical  for  the  successful  development  of  the  capability 
needed  by  the  warfighter. 

The  DOD  monitors  and  controls  system  technical  development  through 
implementation  of  the  baselines,  audits,  and  technical  reviews  within  an  overarching 
systems  engineering  process  (SEP;  Defense  Acquisition  University  [DAU],  2004). 
Because  of  the  relatively  immature  software  engineering  environment,  significantly 
more  analyses  and  development  of  the  requirements  are  required.  In  addition,  the 
software  architectural  design  effort  depends  on  in-depth  requirements  analysis,  is 
resource  intensive,  and  must  occur  very  early  in  the  developmental  process. 

Effective  management  and  implementation  of  design  metrics  are  essential  in 
developing  software  that  meets  the  warfighters’  needs.  This  management  and 
metrics  effort  supplements  and  supports  the  system’s  technical  development 
through  the  baselines,  audits  and  technical  reviews. 

There  are  numerous  variations  and  models  of  the  SEP.  This  research  uses 
the  model  depicted  in  Figure  1,  which  illustrates  the  systems  engineering  functions 
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described  throughout  this  paper.  The  concepts  are  transferable  to  the  SEP  “V’ 
model  that  the  DOD  currently  uses. 


Systems  Engineering  Process 


Process 

Inputs 


System  Analysis  and 
Control  (Balance) 


Process 

Outputs 


Figure  1.  Systems  Engineering  Process 


The  DOD  Software-Intensive  System  Development 
Problem  and  Research  Technique 


Problem 

From  a  systems  management  perspective,  the  overarching  problem  is  that 
the  DOD  Acquisition  Management  System  produces  both  successful  and 
unsuccessful  software-intensive  systems.  The  management  oversight,  structure,  and 
discipline  offered  do  not  produce  repeatable  success  in  complex,  software-intensive 
systems  development. 


Primary  Research  Question 

The  problem  identified  above  drives  this  primary  research  question:  Why 
does  the  DOD  Acquisition  Management  System  produce  both  successful  and 
unsuccessful  software-intensive  systems? 
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Secondary  Research  Questions 

I  analyze  the  DOD  software-intensive  system  development  challenge  by 
addressing  these  secondary  research  questions: 

•  Does  the  DOD  acquisition  environment  provide  opportunity  for  variable 
results  in  software-intensive  system  development? 

•  How  does  the  software  engineering  environment  impact  DOD  software 
intensive  system  development? 

•  Is  the  DOD  requirements  development  and  communication  process 
sufficient  for  potential  software  developers? 

•  How  is  the  software-intensive  system  architecture  developed  to  ensure 
warfighter  capabilities  are  designed  and  prioritized? 

DOD  Acquisition  Environment 

At  the  top  level,  there  are  the  three  primary  decision  support  systems  used 
within  the  DOD,  and  the  interaction  within  these  systems  significantly  decides  the 
acquisition  of  products  or  services  (DOD,  2013b).  The  three  systems  are  the  Joint 
Capabilities  Integration  and  Development  System  (JCIDS),  which  provides  the 
acquisition  requirements  documents;  the  Defense  Acquisition  System  (DAS),  which 
provides  the  processes  to  develop  and  acquire  the  needed  products  to  fulfill  the 
requirement;  and  the  Planning,  Programming,  Budgeting,  and  Execution  (PPBE) 
process,  which  is  the  funding  resource  management.  Figure  2  depicts  the  three 
support  systems. 


DoD  Decision  Support  Systems 
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Figure  2.  DOD  Decision  Support  Systems 

(from  DOD,  2013b) 
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Software-intensive  systems  are  most  impacted  by  the  JCIDS  and  the  DAS 
Decision  Support  Systems,  and  the  PPBE  process  has  no  particularly  unique  impact 
on  software  intensive  systems  development.  This  research,  therefore,  focuses  on 
elements  of  the  JCIDS  and  DAS  systems. 

Requirements  Generation 

The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  was 
designed  to  assess  capability  requirements  and  associated  capability  gaps  and  risks 
(Chairman  of  the  Joint  Chiefs  of  Staff  [CJCS],  2012,  p.  A-1 ).  Capability  gaps  may  be 
identified  in  one  or  more  of  the  following  areas:  Doctrine,  Organization,  Training, 
Materiel,  Leadership  Policy  and  Education,  Personnel,  Facilities,  and  Policy 
(DOTMLPF-P).  Materiel-related  capability  gaps  become  the  basis  for  the 
requirements  process  that  drives  the  acquisition  community  to  develop  and  acquire 
platforms  designed  to  bridge  all  or  part  of  the  identified  gap.  JCIDS  is  designed  to  be 
an  iterative  process,  beginning  with  a  validated  Initial  capabilities  Document  (ICD), 
triggering  the  acquisition  community  to  begin  an  Analysis  of  Alternatives  (AoA)  on 
candidate  systems  that  potentially  address  the  capability  need.  The  Capabilities 
Design  Document  (CDD)  refines  and  adds  necessary  detail  to  support  the  technical 
design  of  the  system  sought.  The  final  document  in  the  series  is  the  Capabilities 
Production  Document  (CPD),  which  further  refines  the  user  requirements  and  adds 
detail  supporting  the  production  planning  for  the  system.  Although  JCIDS  is 
designed  to  refine  well-defined  requirements,  there  is  clearly  an  opportunity  for 
requirements  creep  with  this  iterative  user  requirements  process. 

After  the  user  community  completes  each  JCIDS  iteration,  the 
program/project/product  manager  (PM)  or  materiel  developer  is  prompted  into 
action.  As  stated,  the  ICD  prompts  an  AoA  identifying  the  possible  systems  that 
could  be  procured  or  developed  to  meet  the  capability  need.  The  CDD  is  a  key 
document  in  the  requirements  generation  cycle  and  is  the  user  community’s  primary 
input  for  the  PM’s  development  of  the  performance  specification  for  the  Request  for 
Proposal  (RFP).  The  CPD  is  the  user’s  key  document  for  driving  production 
decisions,  and  the  PM’s  production  strategy  is  significantly  influenced  by  the  CPD. 

One  of  the  PM’s  most  critical  functions  is  developing  the  performance 
specification  for  inclusion  in  the  RFP.  This  requires  the  PM  team  to  translate  the 
user-stated  needs  from  capabilities- based  language  to  performance- based  language 
that  is  used  to  drive  the  design  efforts  of  potential  system  developers,  usually 
contractors.  This  is  critical  because  the  RFP  is  the  basis  for  the  potential  contractors’ 
proposals  containing  the  estimated  cost,  schedule,  and  technical  performance  they 
plan  to  achieve.  The  submitted  proposals  are  evaluated  and  compared  during  the 
labor-intensive  source  selection  process,  resulting  in  a  contract  award  based  on 
proposal  merit.  If  the  performance  specification  is  incomplete,  vaguely  stated,  or 
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misunderstood,  then  the  source  selection  process  and  contract  award  is  based  on 
incorrect  proposals  and  the  effort  is  significantly  wasted. 

The  selected  contractor  accepts  the  terms  of  the  contract  based  on  the  assumptions 
and  estimates  contained  in  the  proposal.  To  develop  the  proposal,  the  contractor 
translates  the  PM’s  performance  specification  into  a  basic  detailed  specification  so 
that  the  scope  of  work  can  be  estimated  for  the  proposed  cost  and  schedule. 
Correcting  these  performance  specification  deficiencies  later  puts  the  Government  at 
a  significant  disadvantage  as  the  contract  has  been  awarded  and  necessary 
changes  to  the  contract  are  negotiated  without  competition.  Changes,  additions,  or 
even  clarifications  to  the  performance  specification  after  contract  award  are  likely  to 
impact  the  terms  of  the  contract,  resulting  in  a  negative  impact  to  the  cost,  schedule, 
or  performance  of  the  desired  system. 

The  Defense  Acquisition  System 

The  DOD  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management 
System  (known  as  the  Horse  Blanket)  is  the  framework  for  control  and  management 
of  DOD  systems  development,  based  on  the  SEP.  As  partially  depicted  in  Figure  3, 
the  model  depicted  features  development  phases  that  define  activities,  and 
milestones  that  serve  as  control  and  decision  points.  These  phases  and  milestones 
are  established  very  early  in  the  development  cycle  using  the  information  available 
during  early  Materiel  Solution  Analysis  (MSA),  which  is  obviously  very  limited. 
Overwhelmingly,  the  PM  responsible  for  establishing  this  strategy  is  not  the 
individual  responsible  for  executing  it.  Funding  requirements,  including  amount,  type, 
and  period  of  execution,  are  established  in  the  Program  Objective  Memorandum 
(POM)  submission  and  a  Congressionally-approved  funding  profile  is  established  for 
the  entire  acquisition  strategy  within  the  PPBE  process.  At  this  point,  the  schedule 
becomes  very  rigid  as  Congress  must  approve  significant  changes  to  the  funding 
profile,  including  when  the  funding  is  to  be  executed.  Although  there  are  obviously 
known  and  unknown  risks  associated  with  an  acquisition  strategy  formulated  this 
early,  there  is  no  provision  for  a  management  reserve  of  funding  to  address  these 
risks. 
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Figure  3.  Defense  Acquisition  Management  System 

(from  DOD,  2013b,  p.  9) 


The  Interim  DOD  Instruction  5000.02,  dated  November  26,  2013,  shows 
alternate  versions  of  the  DAS  phases  and  milestones  (see  Figure  3)  that  attempt  to 
address  the  impact  that  software  imparts  on  the  development  process.  The  interim 
instruction  depicts  the  following  variants  of  the  model:  Defense  Unique  Software 
Intensive  Program;  Incrementally  Fielded  Software  Intensive  Program;  Hybrid 
Program  A  (Hardware  dominant);  and  Hybrid  Program  B  (Software  Dominant) 
models  (DOD,  2013b,  pp.  10-14).  The  new  models  indicate  an  understanding  that 
software  impacts  the  system  development  process  differently  than  typical  hardware 
systems  do.  As  these  are  all  newly  developed,  their  impact  on  future  development  is 
unknown. 


Performance  Specifications  and  the  Work  Breakdown 
Structure 

Since  the  implementation  of  acquisition  reform  in  the  nineties,  detailed 
specifications  have  been  replaced  with  performance  specifications  in  order  to 
leverage  the  considerable  experience  and  expertise  available  in  the  defense 
contractor  base.  In  most  hardware-centric  engineering  disciplines,  the  expertise  that 
the  DOD  seeks  to  leverage  includes  a  mature  engineering  environment  in  which 
materials,  standards,  tools,  techniques,  and  processes  are  widely  accepted  and 
implemented  by  industry  leaders.  This  engineering  maturity  helps  to  account  for 
derived  and  implied  requirements  not  explicitly  stated  in  the  performance 
specification.  Three  levels  of  the  work  breakdown  structure  (WBS)  may  provide 
sufficient  detail  for  vendors  to  develop  a  desired  system  in  a  mature  engineering 
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environment,  such  as  the  automotive  field.  For  example,  an  automotive  design  that 
provides  for  easy  replacement  of  wear-out  items  such  as  tires,  filters,  belts,  and 
batteries  obviously  provides  sustainability  performance  that  is  absolutely  required. 
Most  performance  specifications  do  not  explicitly  address  this  capability  as  they 
would  be  automatically  considered  by  any  competent  provider  within  the  mature 
automotive  engineering  environment. 

The  Department  of  Defense  Handbook :  Work  Breakdown  Structures  for 
Defense  Materiel  Items  (MIL-HDBK-881  A),  recommends  a  minimum  of  three  levels 
be  developed  before  handoff  to  a  contractor  (DOD,  2005).  If  a  program  is  expected 
to  be  high-cost  or  high-risk,  it  is  critical  to  define  the  system  at  a  lower  level  of  the 
WBS  (DOD,  2005,  p.  3).  Complex  weapon  systems  are  nearly  always  high-cost,  and 
the  complex  software  development  that  these  systems  require  almost  always  means 
that  the  development  effort  is  high-risk  as  well.  The  WBS  and  performance 
specification  must,  consequently,  be  significantly  more  developed  to  provide  the 
software  engineer  enough  information  and  insight  to  accurately  estimate  the  level  of 
effort  needed — cost  and  schedule — and  to  actually  produce  the  capabilities  needed 
by  the  warfighter.  Contracts  resulting  from  proposals  that  are  based  on 
underdeveloped,  vague,  or  missing  requirements  typically  result  in  catastrophic  cost 
and  schedule  growth  as  the  true  demands  of  the  software  development  effort  are 
discovered  only  after  contract  award. 

The  WBS  provides  the  basis  for  the  vendors’  performance  specification.  It  is 
also  a  powerful  communications  medium  with  potential  contractors,  as  its  upper 
levels  provide  a  functional  system  breakdown  structure  from  the  DOD’s  perspective. 
The  same  WBS  continues  to  be  developed  by  the  contractor,  eventually  providing 
the  detailed  breakdown  structure:  the  basis  for  the  cost  and  scheduling  estimates 
provided  in  the  proposals  and  used  in  the  Earned  Value  Management  (EVM)  metrics 
during  execution. 

Technology  Readiness  Assessment  and  Risk  Management 

Another  important  management  aspect  is  addressing  the  readiness  of  the  key 
technologies  for  successful  development  and  deployment.  A  Technology  Readiness 
Assessment  (TRA)  is  required  for  most  Major  DOD  Acquisition  Programs  (MDAPs; 
DOD,  201 1 ,  p.  1-1)  The  purpose  for  conducting  a  TRA  is  to  address  the  risk  of 
attempting  to  develop  a  system  with  a  key  technology  that  is  too  immature  to 
successfully  deploy  the  system  when  needed  by  the  warfighter.  To  benchmark  the 
assessment,  Technology  Readiness  Levels  (TRLs)  have  been  developed  in  a  nine- 
level  model,  with  a  goal  of  ensuring  that  a  system’s  key  technologies  achieve  at 
least  a  TRL  level  6  to  reduce  risk  to  an  acceptable  level. 
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There  are  software  TRLs  established,  and  level  6  is  defined  as  “Module 
and/or  subsystem  validation  in  a  relevant  end-to-end  environment.”  The  level  6 
description  specifies  the  “level  at  which  the  engineering  feasibility  of  a  software 
technology  is  demonstrated.  This  level  extends  the  laboratory  prototype 
implementations  on  full-scale  realistic  problems  in  which  the  software  technology  is 
partially  integrated  with  existing  hardware/software  systems”  (Blanchette,  Albert,  & 
Garcia-Miller,  2010,  p.  35). 

The  software  TRL  level  6  description  presents  several  problems  in  performing 
the  TRA  on  a  software-intensive  system.  Weapon  system  software  is  typically 
engineered  from  scratch  with  few  reused  elements,  which  means  that  there  is  very 
little  to  nothing  on  which  to  perform  the  assessment.  There  will  likely  be  software 
developed  for  similar  systems  that  would  meet  the  level  6  description,  but  assessing 
like-software  built  for  another  system  will  not  significantly  reduce  the  software 
technology  risk  of  the  proposed  system.  For  example,  the  F-35  is  built  by  the  same 
manufacturer  as  the  F-22,  and  they  are  both  high-performance  military  aircraft  with 
different  but  overlapping  missions.  Yet  the  F-35  is  experiencing  more  software 
development  problems  than  its  predecessor  and  already  has  three  times  more 
software  than  the  F-22  (Hagan  et  al.,  2013,  p.  26). 

Software  TRLs  do  not  appear  to  be  providing  the  same  type  readiness 
indicator  as  hardware-related  TRLs,  leaving  software  technology  risks  substantially 
unknown.  In  a  2010  U.S.  Army  workshop  report  from  the  Software  Engineering 
Institute  (SEI),  the  participants  noted  that  “though  marginally  useful,  these  efforts 
have  only  confirmed  for  the  participants  the  futility  of  continuing  to  base  [technology] 
readiness  decisions  for  software  aspects  of  systems  on  the  DoD  software  TRLs” 
(Blanchette  et  al.,  2010,  p.  2).  The  software  TRLs  clearly  do  not  seem  to  be  effective 
at  reducing  risk  for  the  TRA. 

To  help  with  early  risk  management  in  lieu  of  effective  software  TRLs,  a 
software  developer  maturity  assessment  is  mandated  for  most  software-intensive 
systems,  through  attaining  level  3  in  the  SEI’s  Capability  Maturity  Model  Integrated 
(CMMI),  or  equivalent,  assessment  methodology  (DOD,  2013a,  p.  92)  The  concept 
recognizes  that  the  software  build  is  a  product  of  the  process,  and  more  mature 
organizations — those  with  successful  past  performance,  demonstrated  engineering 
discipline,  stable  development  staffs,  and  effective  management  structures — reduce 
system  development  risk. 

SEI  also  has  the  Software  Acquisition  Capability  Maturity  Model®  (SA-CMM), 
which  is  designed  to  evaluate  the  maturity  of  software  acquiring  organizations  such 
as  the  DOD’s  software-intensive  system  PM  offices  (Cooper  &  Fisher,  2002).  The 
SA-CMM  is  also  a  five-level  model,  similar  to  the  CMMI.  The  DOD  currently  has  no 
requirement  for  PM  offices  to  undergo  an  evaluation  or  achieve  any  SA-CMM  level, 
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but  the  maturity  of  the  team  responsible  for  communicating  the  system  requirements 
and  managing  the  development  has  an  impact  on  risk. 

Findings  Summary 

In  summary,  the  DOD  acquisition  environment  features  a  requirements  flow- 
down  process  that  involves  user-stated  capabilities-based  requirements  translated  to 
performance-based  requirements,  then  translated  to  the  detailed  design 
specifications.  This  requirements  translation  process  is  the  basis  for  the  resource¬ 
intensive  source  selection  and  binding  contracting  processes,  which  are  critical  for 
accurate  cost  and  schedule  estimates.  Although  DOD  acquisition  is  based  on  the 
event-driven  SEP,  the  schedule  becomes  rigid  very  early  in  the  process  when  time- 
specific  funding  is  attached.  The  subsequent  system  PMs  are  charged  with 
managing  the  cost,  schedule,  and  performance  set  by  the  initial  PM  with  no  funding 
provided  for  managing  the  associated  risk.  To  reduce  risk,  PMs  are  directed  to 
perform  TRAs  early  in  the  process,  with  a  goal  of  achieving  at  least  TRL  6  on  key 
technologies.  Software  TRLs  do  not  appear  to  be  effective,  and  software  developer 
maturity  assessments  are  conducted  to  help  reduce  system  development  risk.  The 
latest  Interim  DOD  Instruction  5000.02  (DOD,  2013a)  depicts  newer  phases  and 
milestone  models  that  attempt  to  address  the  differences  that  software  development 
causes  in  the  management  of  the  DAS. 

DOD  Acquisition  Environment  Analysis 

Does  the  DOD  acquisition  environment  provide  opportunity  for  variable 
results  in  software-intensive  system  development? 

The  DOD  acquisition  environment  appears  to  remain  vulnerable  to  significant 
variability  when  developing  software-intensive  systems,  similar  to  the  problems 
currently  plaguing  the  F-35  JSF  program.  Although  the  new  phases  and  milestones 
models  address  the  software  component  development,  other  critical  management 
functions  remain  unchanged.  Requirements  generation,  performance  specification 
development,  RFP,  source  selection,  and  contracting  processes  have  yet  to  adapt  to 
the  unique  challenges  presented  when  managing  software-intensive  system 
development.  Early  program  risk  management  assesses  key  technology  readiness, 
but  the  software  TRLs  are  ineffective  for  predicting  software  development  risk. 
Evaluating  the  software  developer’s  maturity  helps  reduce  some  risk  but  fails  to 
include  the  critical  DOD  entities  in  any  maturity  assessment. 

The  challenges  software  components  present  within  a  DOD  system 
development  effort  are  being  recognized,  and  the  software-oriented  phases  and 
milestone  models  appearing  in  the  interim  DODI  5000.02  represent  a  beginning  to 
addressing  those  challenges.  The  new  models  depict  the  iterative  nature  of  the 
software  builds  and  how  the  phases  and  milestones  may  adapt  depending  on  how 
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software  intensive  the  system  may  be.  The  other  critical  functions  and  processes 
within  the  Defense  Acquisition  Management  System  do  not  appear  to  be  adapting  to 
the  software  challenges.  The  requirements  generation  system  is  key  and  will  be 
addressed  in  detail  later  in  this  research,  but  it  doesn’t  appear  to  have  changed 
significantly.  As  discussed  earlier,  those  requirements  generated  through  the  system 
are  the  drivers  and  the  basis  for  other  critical,  early  systems  analysis  and 
communication.  The  performance  specification,  RFP,  contractor  proposal 
generation,  source  selection,  and  contracting  processes  remain  essentially 
unchanged. 

Early  risk  management  through  the  TRA  and  achieving  a  desired  TRL  is 
ineffective  for  the  software  component.  Assessing  the  contractor  (software 
developer)  maturity  through  CMMI  or  equivalent  evaluation  appears  to  be  effective  in 
reducing  the  developer  risk  but  does  not  address  the  DOD  acquisition  community 
maturity.  As  the  software  developer  is  significantly  dependent  on  the  Government’s 
ability  to  effectively  generate  and  clearly  communicate  a  comprehensive  set  of 
requirements,  quality  attributes,  and  critical  design  elements,  assessing  just  the 
developer’s  maturity  addresses  only  part  of  the  risk. 

Software  Engineering  Environment 

Software  Engineering 

The  software  engineering  environment  is  not  mature,  especially  when 
compared  to  hardware-centric  engineering  environments.  Dr.  Philippe  Kruchten 
(2005)  of  the  University  of  British  Columbia  remarks,  “We  haven’t  found  the 
fundamental  laws  of  software  that  would  play  the  role  that  the  fundamental  laws  of 
physics  play  for  other  engineering  disciplines”  (p.  17).  Software  engineering  is 
significantly  unbounded  because  there  are  no  physical  laws  that  help  define 
environments.  There  is  significant  evidence  for  software  engineering  immaturity,  and 
it  is  nearly  impossible  to  find  widely  accepted,  industry-wide  development  standards, 
protocols,  architectures,  or  formats.  There  is  no  dominant  programming  language, 
design  and  development  process,  standard  architectures,  or  software  engineering 
tools,  which  means  that  reusable  modules  and  components  rapidly  become 
obsolete.  All  of  these  combine  to  make  it  nearly  impossible  to  institute  a  widely 
accepted  software  reuse  repository.  Without  significant  software  architecture  and 
code  reuse  in  developing  software-intensive  weapon  systems,  each  development 
process  essentially  starts  from  scratch.  This  fact  is  one  of  the  main  reasons  that  the 
TRA  and  the  software  TRLs  are  ineffective  in  predicting  software  development  risk 
(Naegle  &  Petross,  2007). 

The  software  engineering  state-of-the-practice  currently  is  wholly  dependent 
on  the  requirements  that  are  passed  to  the  software  development  team.  From  the 
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requirements,  a  software  architecture  is  designed,  and  the  requirements  “flow  down” 
through  that  architecture  to  the  individual  modules  and  computer  software  units  that 
are  to  be  constructed.  The  software  build  focuses  on  the  requirements  that  flowed 
down  to  that  level  and  the  integration  required  for  functionality.  The  standards, 
protocols,  formats,  languages,  and  tools  used  for  the  build  will  likely  be  unique  to  the 
contractor  developing  the  software,  and  will  most  certainly  not  be  universally 
accepted  or  recognized  across  the  software  industry. 

The  software  architectural  design  is  the  basis  for  all  of  the  current  and  future 
system  performance  that  the  system  will  achieve,  and  the  current  state-of-the- 
practice  in  software  engineering  has  each  project  design  a  unique  architecture.  Like 
hardware,  the  software  design  will  significantly  impact  system  attributes  that  are 
important  to  the  warfighter,  including  maintainability,  upgradability,  interoperability, 
reliability,  safety,  and  security.  Most  hardware-oriented  engineering  environments 
address  these  critical  areas  through  widely  accepted  industry  standards.  For 
example,  all  DOD  ground  combat  vehicles  use  a  24  volt,  direct  current,  negative 
ground  electrical  system.  Any  current  or  future  subsystem  requiring  vehicle  power 
will  automatically  be  designed  to  operate  using  those  industry-wide  electrical  power 
standards. 

Comparison  to  Mature  Engineering 

The  software  engineering  environment  is  in  stark  contrast  to  even  our  most 
advanced  hardware-centric  engineering  environments.  For  example,  in  the 
automotive  engineering  field,  a  design  that  provides  for  easy  replacement  of  wear- 
out  items  such  as  tires,  filters,  belts,  and  batteries  obviously  provides  sustainability 
performance  that  is  absolutely  required.  This  engineering  maturity  helps  account  for 
derived  and  implied  requirements  not  explicitly  stated  in  the  performance 
specification.  Most  performance  specifications  do  not  explicitly  address  this 
capability  because  they  would  be  automatically  considered  by  any  competent 
provider  within  the  mature  automotive  engineering  environment.  A  mature 
engineering  environment  includes  design  elements  and  industry-wide  standards, 
processes,  materials,  and  techniques  to  which  we  have  grown  to  expect.  A 
significant  problem  will  exist  if  we  expect  the  software  engineering  environment  to 
perform  the  same  way  as  other,  more  mature  engineering  fields  (Naegle  &  Petross, 
2007). 

As  the  example  above  illustrates,  system  sustainability  elements  are  often 
standardized  across  hardware-oriented  engineering  environments.  Without  the 
engineering  maturity,  software  sustainability  performance  and  expectations  must  be 
specified  as  part  of  the  requirements  generation  process.  The  capabilities-based 
user  requirements  and  performance-based  acquisition  requirements  are  specifically 
not  designed  to  provide  that  level  of  specificity. 
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Findings  Summary 

With  software,  virtually  all  of  the  performance  and  quality  attributes  developed 
come  directly  from  the  requirements  received,  and  the  immature  software 
engineering  environment  will  likely  not  compensate  for  any  desired  performance, 
such  as  system  sustainability,  that  is  not  clearly  specified  in  a  requirement.  Unlike 
hardware-oriented  engineering  environments,  where  the  widely  accepted  industry 
standards  will  be  employed  whether  or  not  they  are  specified,  with  software,  you  get 
what  you  specify  and  very  little  else. 

The  software  architectural  designs  suffer  from  the  immature  engineering 
environment  as  well.  Each  software  design  is  unique  and  driven  by  the  requirements 
received  with  no  industry-standard  architectures  available.  All  current  and  future 
system  attributes  impacted  by  the  architecture  must  be  communicated  to  the 
software  design  staff  to  ensure  they  are  considered  in  the  design  process. 

Software  Engineering  Environment  Analysis 

How  does  the  software  engineering  environment  impact  DOD  software¬ 
intensive  system  development? 

As  illustrated  in  the  previous  section,  the  lack  of  software  engineering  maturity 
impacts  both  requirements  development  and  design  of  the  architecture.  To 
compensate  for  the  relative  immaturity  of  the  software  engineering  environment,  the 
DOD  must  conduct  significantly  more  in-depth  requirements  analysis  and  provide 
potential  software  developers  detailed  performance  specifications  in  all  areas  of 
software  performance  and  sustainability.  This  is  a  significantly  different  mind-set 
than  the  hardware-dominated  systems  acquisition  of  the  past. 

In  addition  to  the  performance  requirements,  software  architectures  must  be 
similarly  shaped  to  include  system  attributes  expected  by  the  warfighter.  Many  DOD 
user  representatives  and  acquisition  professionals  have  grown  accustom  to  the 
engineering  maturity  levels  offered  by  the  hardware-oriented  systems  that 
dominated  past  acquisitions.  Providing  the  system  requirements  in  the  same  fashion 
may  not  drive  the  architecture  for  needed  attributes.  As  demonstrated  by  the  F-35 
JSF  redesign  problems,  changing  software  architectures  during  the  development 
cycle  will  likely  be  costly  in  terms  of  schedule  and  funding. 
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DOD  Acquisition  Environment:  Impact  on  Software 
Development  and  Quality  Attributes 

DOD  Requirements  Generation  Process 

The  DOD  requirements  generation  process  was  described  earlier  as  part  of 
the  DOD  acquisition  environment  and  consists  of  three  major  processes:  user¬ 
generated  requirements  in  the  form  of  capability  needs  using  the  JCIDS;  PM- 
generated  requirements  in  the  form  of  performance  specifications ;  and  finally, 
contractor-generated  detailed  specifications,  developed  generally  in  that  order.  Two 
major  requirements  language  interpretations  are  required  to  get  from  the  warfighters’ 
needs  to  the  system  built  to  meet  those  needs,  leaving  significant  opportunity  for 
misinterpretation,  omission,  and  misunderstanding  of  weakly  articulated  and  vaguely 
stated  language.  To  do  this  effectively,  the  PM  must  accurately  interpret  user 
capability  language  (as  an  example,  warfighter  requires  the  capability  to  ...  in  all 
mission  environments)  and  translate  that  into  performance  language  (system  shall 
achieve  xxx  performance  ...  in  these  specific  conditions,  for  example).  The 
contractor  then  translates  the  performance  language  into  the  system  build-details 
that  meet  or  exceed  the  performance  specified. 

The  importance  of  system  software  requirements  development  to  the 
potential  success  of  software-intensive  systems  development  cannot  be  overstated. 
Underdeveloped,  vaguely  articulated,  ill-defined  software  requirements  elicitation 
has  been  linked  to  poor  cost  and  schedule  estimations,  resulting  in  disastrous  cost 
and  schedule  overruns  such  as  what  the  F-35  JSF  is  currently  experiencing.  In 
addition,  the  resulting  products  have  been  lacking  important  functionality,  are 
unreliable,  and  have  been  costly  and  difficult  to  effectively  sustain  (Naegle,  2006). 

Systems  Engineering  Process 

Using  the  SEP  approach,  the  explicit  user  capabilities  requirements  specified 
in  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  provides  the 
input  for  system  requirements  analyses.  These  analyses  are  intended  to  illuminate 
all  system-stated,  -derived,  and  -implied  requirements  and  quality  attributes 
necessary  to  achieve  the  capabilities  needed  by  the  warfighter.  The  WBS  is  a 
methodology  for  defining  ever-increasing  levels  of  performance  specificity  using  the 
SEP  to  guide  the  development  of  each  successive  layer  (DOD,  2005,  pp.  1-5). 

Just  as  it  supports  hardware  development,  the  Systems  Engineering 
Process  (SEP)  is  essential  in  the  development  of  software  design.  In 
software  development,  good  quality  and  predictable  results  are 
paramount  goals  in  creating  the  specified  warfighter  capabilities  within 
cost  and  schedule  constraints.  To  accomplish  those  goals,  we  examine 
the  methods,  tools  and  processes  the  software  developer  uses  in 
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building  the  software  with  the  intent  of  attaining  a  product  that  provides 
all  of  the  necessary  functionality  and  is  supportable,  efficient,  reliable 
and  easy  to  upgrade.  (Naegle  &  Petross,  2007,  pp.  14,  15) 

Work  Breakdown  Structure 

In  previous  research  conducted  by  Ms.  Diana  Petross  and  me,  we  addressed 

the  WBS  in  detail: 

The  Government’s  requirements  and  specifications  for  a  new  weapon 
system  are  detailed  in  the  RFP;  this  includes  a  Government-produced 
Work  Breakdown  Structure  (WBS)  (composed  of  at  least  three  levels). 
This  is  known  as  the  Program  WBS  and  is  handed  off  to  the  contractor 
to  develop  a  WBS  that  defines  the  level  of  detail  required  for  product 
development.  This  contractor-generated  product  will  ensure  the  system 
developer  understands  the  program  objectives  and  the  products  to  be 
delivered  in  performance  of  the  contract.  The  WBS  details  the 
functionality  and  performance  of  the  system  and  provides  a  baseline  to 
track  performance  against  cost  and  schedule.  For  most  hardware¬ 
centric  programs,  a  WBS  for  the  top  three  levels  of  the  system  under 
development  is  usually  enough  detail  to  manage  the  program. 

Because  of  the  volatile  nature  of  software  development,  immature 
software  engineering  environment,  and  the  potential  impact  to  cost, 
schedule  and  risk,  the  WBS  for  software-intensive  programs  needs  to 
be  developed  down  to  Level  5  or  lower — including  system-of-systems 
(SOS)  and  net-centric  systems  development. 

Level  1  of  the  WBS  describes  the  entire  project.  If  the  program  is  a 
systems-of-systems  (SOS)  project,  Level  I  becomes  that  overarching 
system.  For  instance,  the  Army  Future  Combat  System  (FCS)  had  a 
number  of  platforms  that  were  segments  of  the  total  system.  Each 
platform  becomes  a  major  segment  of  that  product  (Level  2);  the 
software  development  would  then  be  broken  down  to  Level  6,  which 
identifies  software-configuration  items. 

Using  the  FCS  as  an  example,  Level  1  describes  the  overall  FCS 
concept  and  environment.  Level  2  details  the  major  product  segments 
of  the  SOS.  In  our  example  of  the  FCS,  the  Level  2  would  be  the 
platforms,  i.e. ,  infantry-carrier  vehicles,  command  vehicles,  mounted 
combat  systems,  etc. 

Level  3  defines  the  major  components  (or  subsets)  of  Level  2.  For 
software  development,  decomposition  of  the  software  WBS  to  the 
lowest  component  is  critical  for  the  developer  to  fully  comprehend  the 
detailed  level  of  effort  required  to  design  and  develop  effective 
systems.  Under  the  FCS  scenario,  Level  3  would  be  one  of  the 
subsystems  onboard  the  manned  systems,  e.g.,  the  fire-control 
systems  and  environmental-control  systems.  It  is  clear  that  WBS 
definition  to  this  level  provides  only  a  very  top-level  insight  to  the 
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system  being  developed;  thus,  for  the  software-intensive  system,  the 
WBS  fails  to  convey  enough  information  for  the  contractor  to  propose  a 
realistic  cost  and  schedule  estimate.  Too  much  of  the  software 
development  work  is  hidden  at  this  level. 

Level  4  becomes  a  breakout  of  the  component  parts  of  the  subsystem. 
Using  a  manned  vehicle  in  the  FCS  program,  Level  5  of  the  WBS 
would  identify  the  component  functions  for  the  fire-control  system:  for 
example,  detect  the  target,  aim  at  the  target  and  fire  the  munitions.  The 
software  build-set  would  support  the  functionality  of  that  component 
within  the  subsystem.  Again,  using  the  FCS  as  the  overarching 
program,  Level  6  is  a  sum  of  software  items  (Si’s)  which  satisfy  a 
required  function  and  are  designated  for  configuration  management.  If 
the  software  requirements  or  attributes  are  well  defined,  the  result  is  a 
product  that  is  properly  designed  to  functionally  perform  to  the  users’ 
requirements.  Further  development  below  Level  6  may  be  necessary 
to  adequately  convey  the  derived  and  implied  requirements  to  the 
software  developer.  (Naegle  &  Petross,  2007,  pp.  13,  14) 

Software  Engineering  Maturity  Impact  on  Requirements 
Generation 

The  immature  software  engineering  environment,  discussed  earlier,  can  be 
compensated  for  only  by  a  requirements  generation  system  that  does  not  leave  any 
gaps  in  performance  or  quality  attributes  needed.  Having  all  of  the  requirements 
clearly  communicated  is  critical,  but  the  software  engineer  must  also  understand  the 
requirements  in  context.  For  example,  the  Ml  A2  Abrams  main  battle  tank  uses 
numerous  inputs  for  precisely  engaging  threat  targets.  Several  such  inputs  are 
essential  for  any  acceptable  probability  of  hitting  the  desired  target,  including  target 
acquisition  (finding  the  target),  location  (azimuth  and  range),  aiming/tracking,  and 
firing  the  projectile.  To  increase  accuracy,  several  other  systems  are  employed  that 
enhance  one  or  more  of  the  essential  functions,  including  cross-wind  sensor, 
temperature  sensor,  muzzle-reference  system,  and  others  (Naegle,  2007,  p.  18). 
Both  essential  and  enhancing  features  are  communicated  to  the  system  and 
software  developers  as  requirements  and,  as  such,  appear  to  have  equal  weight. 

The  critical  difference  between  “essential”  and  “enhancing”  may  not  be  clear  to  the 
software  development  team,  which  may  result  in  a  poorly  performing  and  possibly 
dangerous  design.  For  example,  as  actually  happened  in  Ml  Abrams  testing,  the 
temperature  sensor  malfunctioned  and  was  reporting  a  temperature  of  2,000 
degrees  Fahrenheit,  well  outside  of  the  expected  range,  creating  an  “exception.”  The 
software  engineers  had  designed  the  software  to  abort  firing  in  the  event  of  any 
exception  and,  of  course,  the  warfighter  needs  the  system  to  have  the  ability  to  fight 
if  any  or  all  of  the  enhancing  functions  fail.  The  distinction  needs  to  be  made  clear, 
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but  there  is  no  definitive  method  for  identifying  requirements  as  system  “essential”  or 
“enhancing.” 

System  Operational  Context 

To  gain  some  insight  into  the  operational  environments  that  the  system  is 
expected  to  operate  within,  the  DOD  provides  an  Operational  Mode 
Summary/Mission  Profile  (OMS/MP).  The  OMS/MP  provides  some  basic  insight  into 
the  operational  profile,  threat  profile,  environmental  profile,  and  the  terrain/sea 
state/undersea/air  environment  profile,  which  adds  some  context  to  the 
requirements,  but  is  not  usually  scenario  based.  It  typically  lacks  sustainability 
activities,  interoperability  profiles,  system  life-cycle  profiles,  planned  or  anticipated 
upgrades,  or  operation  in  stressful,  degraded,  or  emergency  situations. 

The  Joint  Light  Tactical  Vehicle  (JLTV),  replacing  the  High  Mobility  Multi¬ 
purpose  Wheeled  Vehicle  (HMMWV)  family  of  vehicles,  is  a  multi-mission  platform. 
The  JLTV  program  OMS/MP  (version  3.3)  dated  January  12,  2012.  Under  paragraph 
1 .2,  Document  Overview,  it  explains  what  the  JLTV  OMS/MP  defines,  including 

•  Expected  operational  modes 

•  Full  spectrum  operations,  operational  themes,  and  elements  of  the 
operational  terms  (offense,  defense,  and  stability) 

•  Joint  mission  profiles  and  operational  elements 

•  Terrain  conditions  in  terms  of  mileage,  speed,  and  roughness 

•  Environmental  conditions  (DOD,  2012,  p.  3) 

Obviously,  a  significant  amount  of  mission  and  system  information  is  omitted. 
The  details  regarding  the  following  items  are  not  provided: 

System  mission  configuration  (JLTV  is  a  multi-mission  platform) 

Number  of  personnel 

Personnel  equipment  and  supplies  required 
Cargo/hauling  capacity 
Interoperability  requirements 

o  Communications  and  network  equipment 
o  Situational  awareness  systems 
o  Weapons 

o  Trailers/towed  systems 

o  Special  missions  equipment  (e.g.,  chemical  weapons  detection) 
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o  Electric  power  requirements  for  integrated  equipment 


•  Crew  maintenance  tasks  and  frequency 

•  Survivability  considerations 


As  with  this  JLTV  OMS/MP,  the  documents  typically  do  not  provide  much 
information  on  operations  under  stressful,  degraded,  or  unusual  conditions,  which 
would  provide  critical  design  cues  for  the  software  engineers.  There  is  no 
prioritization  of  the  operational  modes  or  configurations,  nor  identification  of  critical 
and  non-critical  systems. 

The  software  development  team  would  likely  continue  to  be  missing  important 
information  that  they  need  to  adequately  design  the  software  and  to  predict  the 
funding  and  schedule  resources  necessary  to  build  the  software  the  warfighter 
expects.  The  JLTV  OMS/MP  was  specifically  created  for  the  Engineering  and 
Manufacturing  Development  (EMD)  phase.  In  paragraph  1.1,  Purpose,  it  states, 


This  OMS/MP  describes  system  modes,  mission  profiles,  and  usage 
conditions  for  the  JLTV  during  its  operating  life.  When  approved,  it 
supersedes  the  OMS/MP  published  with  the  JLTV  Request  for 
Proposal  (RFP)  in  February  2008,  but  will  not  take  effect  until  JLTV 
EMD  phase  activities.  The  OMS/MP  supports  the  basis  for  essential 
capabilities  described  in  the  JLTV  Capability  Development  Document 
(ODD)  documenting  key  usage  factors  directly  applicable  to  design 
study,  logistical  analyses,  O&S  [operations  and  support]  estimation, 
and  reliability,  availability,  and  maintainability  (RAM)  testing  and 
analyses.  (DOD,  2012  p.  3) 


The  OMS/MP  documents  do  not  typically  provide  any  information  regarding 
system  life-cycle  changes  such  as  pre-planned  product  improvement  (P3I) 
programs,  planned  upgrades  and  technology  refreshments,  future  interoperability 
requirements,  or  plans  for  future  integration  into  tactical  and  logistical  networks. 
These  life-cycle  events,  while  known  or  anticipated,  are  not  effectively 
communicated  to  potential  developers  for  inclusion  in  the  proposal  process  and  are 
often  omitted  from  the  software  system  design. 

The  JLTV  OMS/MP  was  selected  because  the  JLTV  (a  HMMWV 
replacement)  is  a  system  that  is  easy  for  nearly  all  readers  to  comprehend,  and  it  is 
obviously  not  a  software-intensive  system.  The  analysis  of  this  system  is  intended  to 
illustrate  that  the  typical  OMS/MP  provides  only  the  basic  insight  into  even  the  most 
basic  system  to  be  developed,  and  relies  heavily  on  DOD  acquisition  professionals 
to  ensure  that  the  contractor  has  a  sufficient  understanding  of  the  operational  role, 
quality  attributes,  system  life-cycle  changes,  and  expected  operation  in  stressful  or 
degraded  modes  to  adequately  design  and  produce  to  the  warfighter’s  expectations. 
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Impact  on  Software  and  Quality  Attributes  Analysis 

Is  the  DOD  requirements  development  and  communication  process  sufficient 
for  potential  software  developers? 

The  DOD  requirements  generation  process  that  was  purposefully  designed  to 
garner  the  maximum  contractor  innovation  and  flexibility  appears  to  provide  too  little 
information  for  the  software  developer  to  adequately  predict  the  resources 
necessary  to  develop  the  system  software.  It  is  clear  that  the  current  state  of  the 
software  engineering  environment  is  mostly  incapable  of  compensating  for  missing, 
vaguely  stated,  or  weakly  articulated  requirements.  At  the  same  time,  the  current 
DOD  requirements  generation  system  provides  ample  opportunity  to  inadvertently 
omit  requirements  and  to  provide  vaguely  stated  or  weakly  articulated  requirements 
through  the  capabilities-oriented  JCIDS  documents  and  the  performance-based 
specifications  derived  from  them. 

Without  fully  understanding  the  requirements  in  a  detailed  operational 
context,  the  software  design  and  development  effort  and  resources  remain 
significantly  unknown.  The  typical  OMS/MP  provides  some  operational  context  to  the 
requirements,  but  is  not  sufficiently  detailed  to  provide  the  design  drivers  needed  by 
the  software  engineers.  Developing  a  proposal  with  this  limited  information  will  likely 
result  in  a  significantly  underestimated  software  development  effort.  After  contract 
award,  more  operational  details  are  typically  provided  through  program  and  design 
reviews,  and  the  cost  and  schedule  for  the  software  effort  are  likely  to  inflate 
significantly  to  accommodate  the  new  understanding  of  the  requirement  in  a  non¬ 
competitive  environment. 

The  lack  of  operational  context  typically  provided  by  the  Government  during  the  RFP 
process  appears  to  have  significant  negative  impacts  on  the  software  design  for 
reliability  and  maintainability.  The  OMS/MP  documents’  lack  of  information  regarding 
significant  planned  and  anticipated  life-cycle  changes,  system  sustainment  activities 
and  burden,  and  operations  under  unusual  conditions  will  likely  mean  that  the 
system  software  design  will  not  easily  accommodate  known  changes.  There  is  no 
prioritization  of  the  operational  modes  or  configurations  that  would  impact  system 
design  considerations.  This  information  would  also  help  differentiate  critical  systems 
from  enhancing  (non-critical)  systems,  providing  a  priority  in  the  software  design 
effort. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


■  19- 


Software-Intensive  System  Architecture  Development 
Analysis 

How  is  the  software-intensive  system  architecture  developed  to  ensure 
warfighter  capabilities  are  designed  and  prioritized? 

The  DOD  system  architectural  process,  with  all  of  its  tools,  techniques,  and 
discipline,  appears  to  be  ineffective  in  driving  repeatable,  successful  software 
designs.  Within  the  SEP,  there  are  three  DOD  processes  that  drive  the  system 
architecture:  the  requirements  generations  system,  the  WBS,  and  the  OMS/MP. 

This  research  has  previously  analyzed  how  that  process  is  not  effective  in 
providing  repeatable,  effective  software-related  requirements  detail.  The  DOD 
requirements  generation  process  develops  system  requirements  within  the  SEP, 
culminating  in  the  system  performance  specification  and  included  in  the  RFP.  There 
appears  to  be  significant  opportunity  to  omit  requirements,  or  to  provide  vague  or 
weakly  articulated  requirements  through  the  translation  process  from  the  user 
capability-based  requirements,  to  the  PM’s  performance  specification,  and  finally  to 
the  contractor’s  detailed  specification.  This  problem  is  exacerbated  by  the  immature 
software  engineering  environment  described  earlier,  which  is  solely  focused  on 
requirements  as  provided. 

The  process  of  developing  the  WBS  appears  to  be  similarly  flawed  in 
effectively  communicating  the  functional  architecture  to  a  sufficient  level  for  the 
software  developers.  The  requirements  are  developed  in  concert  with  the  system 
WBS,  which  is  a  primary  tool  for  communicating  functional  architecture  from  the 
DOD  perspective.  After  the  DOD  has  completed  its  portion  of  the  WBS,  it  is  handed 
off  to  the  contractor  to  complete  the  effort  by  developing  the  detailed  WBS  to  the 
lowest  level  deemed  necessary  to  code  and  build  the  software  units.  The 
overarching  philosophy  for  both  requirements  generation  and  the  WBS,  in  order  to 
garner  the  maximum  flexibility  and  innovation,  is  purposely  not  to  be  specific.  Due  to 
the  immature  engineering  environment,  the  software  components  need  significantly 
more  specificity  than  the  hardware  counterparts  to  produce  realism  in  the  cost  and 
schedule  provided  in  the  contractor’s  proposal. 

The  operational  context  information  that  the  Government  provides  appears  to 
be  insufficient  for  the  potential  software  developers  to  have  an  understanding  of  the 
requirements  within  the  context  of  the  operational  environment,  constraints,  and  life- 
cycle  events  of  the  proposed  system.  The  OMS/MP  typically  provides  only  a  vague 
understanding  of  the  operational  environment  and  significantly  more  information  is 
required  to  design  and  build  the  system  actually  needed  by  the  warfighter.  This 
additional  information  is  likely  to  be  added  in  program  and  design  reviews  conducted 
after  the  contract  is  awarded,  so  resulting  changes  impacting  the  software 
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development  can  cause  significant  increases  in  the  cost  and  schedule,  all  negotiated 
without  the  advantages  of  a  competitive  environment. 

Conclusions 

The  DOD  acquisition  process  provides  the  environment  for  both  successful 
and  unsuccessful  software-intensive  systems  development.  Specific  elements  of  the 
DOD  acquisition  process  that  contribute  to  the  variable  environment  include  the 
following: 

•  The  DOD  Requirements  Generation  Process.  The  translation 
process  from  JCIDS  capabilities-based  language  to  the  RFP/contract 
performance-based  language,  and  finally  to  the  specification-based 
detailed  language  creates  ample  opportunity  for  misinterpreted 
requirements  to  be  communicated.  This  process  was  designed  to 
garner  innovation  from  mature  engineering  fields  that  leverage  widely 
accepted  materials,  processes,  and  standards — attributes  that  the 
software  engineering  field  does  not  yet  have. 

•  Communicating  Operational  Context.  The  Operational  Mode 
Summary/  Mission  Profile  (OMS/MP)  provides  some  insight  into  a 
system’s  intended  operational  context  but  provides  far  too  little 
information  for  the  complex  software  design  process.  This  lack  of 
detail,  again,  cannot  be  compensated  by  the  immature  software 
engineering  environment  and  so  impacts  software-intensive  systems 
more  than  hardware-centric  ones. 

•  Failure  to  Compensate  for  the  Immature  Software  Engineering 
Environment. 

o  As  demonstrated  by  the  first  two  bullets,  one  of  the  major 
differences  between  successful  and  unsuccessful  software¬ 
intensive  systems  development  is  recognizing  and 
compensating  for  the  immature  software  environment.  The  DOD 
Acquisition  System  policies,  guidelines,  and  controls  do  not 
provide  a  framework  to  ensure  that  essential  software  attributes 
are  sufficiently  revealed  and  effectively  communicated  to  the 
contractors  that  will  design  and  build  the  software  systems. 

•  The  DOD  Acquisition  System. 

o  The  DAS  is  designed  to  leverage  industry  innovation  by 

providing  performance  specifications  that  are  designed  to  allow 
mature  industrial  engineering  environments  to  develop  the  best- 
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value  technologies  that  meet  the  performance  specifications. 
This  is  effective  when  the  engineering  environments  are  mature 
and  can  offer  viable,  mature  technology  alternatives  that  are 
considered  industry-standard.  There  are  insufficient  DAS 
processes  for  recognizing  and  compensating  for  immature 
engineering  environments,  such  as  exists  in  the  software  field. 

o  The  schedule  and  funding  profile  are  initially  set  by  the  first 
system  PM,  and  the  program  depends  significantly  on  how  well 
the  requirements  generation  process  accurately  identified  the 
bulk  of  the  requirements.  Once  funding  is  linked  to  milestones, 
the  program  cost  and  schedule  become  very  rigid,  which 
exacerbates  problems  with  software-intensive  system 
developments  that  have  late  requirements  creep  due  to 
insufficient  understanding  of  the  effort  in  the  proposal 
preparation. 

o  Software  Technology  Readiness  Levels  (TRLs)  are  ineffective  in 
reducing  risks  associated  with  the  system  software 
development.  Because  there  are  few  reusable  software 
components,  limited  industry-wide  standards  for  architecture 
and  supportability,  and  rapidly  emerging  languages,  protocols 
and  tools,  the  software  TRLs,  based  on  past  efforts,  are  not 
reliable  predictors  of  software  readiness. 

o  Software  development  significantly  adds  to  the  system 

development  risk.  The  DAS  is  designed  to  reduce  development 
risk,  but  cannot  eliminate  all  associated  risks.  Some  risk  is 
accepted  with  the  expectation  that  the  PM  team  will  effectively 
manage  those  risks,  yet  there  is  no  funding  management 
reserve  provided  to  do  so.  Any  risk  management  mitigation 
effort  that  involves  funding  has  the  opportunity  to  create  a 
cascade  of  management  actions  resulting  from  funding 
reductions  in  other  planned  and  necessary  activities. 
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Recommendations 

General 

As  part  of  this  research,  I  searched  for  tools,  techniques,  and  procedures  that 
would  address  the  software-intensive  system  development  problems  and  integrate 
well  with  the  Defense  Acquisition  System  (DAS)  while  supporting  the  Systems 
Engineering  Process  (SEP).  The  tools,  techniques,  and  procedures  recommended  in 
this  section  are  not  particularly  new  and  many  programs  may  have  used  some, 
most,  or  all  of  these  in  the  development  of  their  systems.  The  major 
recommendation  is  that  DOD  formalize  and  institute  the  use  of  these  tools, 
techniques,  and  procedures  (or  similar  ones)  for  the  development  of  software¬ 
intensive  systems.  There  would  almost  certainly  be  a  benefit  when  applied  to 
hardware-centric  system  development,  too,  and  certainly  there  would  be  no 
detriment  in  using  them  for  all  complex  system  development. 

One  of  the  findings  of  this  research  was  the  lack  of  a  PM  management 
reserve  fund  to  address  accepted  development  risks,  but  a  significant  policy  and 
political  change  would  be  required  to  provide  a  management  reserve  in  program 
funding.  I  believe  this  course  of  action  to  be  unlikely,  but  the  implementation  of  the 
recommendations  would  significantly  reduce  software-intensive  system 
developmental  volatility  and  risk,  and  reduce  the  need  for  the  management  reserve. 

Each  of  the  tools,  techniques,  and  procedures  are  valuable  in  assisting  the 
systems  development  process,  but  when  used  together,  provide  a  synergistic  effect 
to  the  vital  front-end  analyses  that  directly  impact  the  shortcomings  revealed  in  this 
research.  Implementing  these  tools  does  not  require  any  major  adjustments  to  the 
DAS  or  the  SEP,  and  in  fact  become  major  enablers  for  both. 

Tools,  Techniques,  and  Processes 

The  tools,  techniques,  and  processes  are  briefly  described  below. 

The  Software  Engineering  Institute’s  (SEI’s)  Quality  Attribute 
Workshop  (QAW) 

The  Maintainability,  Upgradability,  Interoperability,  Reliability,  &  Safety 
and  Security  (MUIRS)  analytic  technique 

The  Software  Engineering  Institute’s  Architectural  Tradeoff  Analysis 
Methodology  (ATAMsm) 

The  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA) 
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•  Software  Management  Readiness  Levels  (MgtRL) 

Quality  Attribute  Workshop 

The  QAW  is  primarily  a  method  for  more  fully  developing  system  software 
requirements  and  is  intended  to  provide  stakeholders’  input  about  their  needs  and 
expectations  from  the  software  (Barbacci  et  al.,  2003,  p.  1 ).  As  the  system 
requirements  are  developed,  software  quality  attributes  are  identified  and  become 
the  basis  for  designing  the  software  architecture. 

The  SEI’s  QAW  is  implemented  before  the  software  architecture  has  been 
created  and  is  intended  to  provide  stakeholder  input  about  the  needs  and 
expectations  from  the  software  (Naegle,  2007).  The  QAW  process  provides  a 
vehicle  for  keeping  the  combat  developer  and  user  community  involved  in  the  DOD 
acquisition  process,  which  is  a  key  goal  of  that  process.  In  addition,  the  QAW 
includes  scenario-building  processes  that  are  essential  for  the  software  developer  to 
design  the  software  system  architecture  (Barbacci  et  al.,  2003,  pp.  9-1 1).  These 
scenarios  will  continue  to  be  developed  and  prioritized  after  contract  award  to 
provide  context  to  the  quality  attribute  identified  for  the  system. 

Although  the  QAW  would  certainly  be  useful  after  contract  award,  conducting 
the  workshop  between  combat  developers/users  and  the  program  management 
office  before  issuance  of  the  Request  for  Proposal  (RFP)  would  provide  an  improved 
understanding  of  the  requirements,  enhance  the  performance-specification 
preparation,  and  improve  the  ability  of  the  prospective  contractors  to  accurately 
propose  the  cost  and  schedule.  This  approach  would  support  the  goals  of  the 
System  Requirements  Review  (SRR),  which  is  designed  to  ascertain  whether  all 
derived  and  implied  requirements  have  been  sufficiently  defined  (Naegle  &  Petross, 
2007,  pp.  5,  6). 

Primary  Software  Acquisition  Problem  Area  Addressed 

The  QAW  process  is  primarily  designed  to  more  fully  develop  system 
software  requirements  so  that  the  Government  RFP  is  clearer  to  potential 
contractors.  In  turn,  the  resulting  proposals  should  be  more  accurate  and  realistic, 
reducing  requirements  and  project  scope  creep. 

Maintainability,  Upgradability,  Interoperability/Interfaces, 

Reliability,  and  Safety/Security  Analytic  Technique 

The  MUIRS  analytic  technique  is  designed  to  provide  a  framework  for  better 
understanding  of  essential  supportability  and  safety/security  aspects  that  the 
warfighter  needs  and  expects  but  often  doesn’t  communicate  clearly  with  the 
capabilities-based  JCIDS  documents.  This  analytic  technique  helps  compensate  for 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-24- 


the  immature  software  engineering  environment  as  the  MUIRS  analysis  illuminates 
the  derived  and  implied  requirements  that  the  immature  environment  cannot. 

Much  of  the  software  supportability  and  safety/security  performance  that 
typically  lacks  consideration  and  is  not  routinely  addressed  in  the  software 
engineering  environment  can  be  captured  through  development  and  analysis  of  the 
MUIRS  elements.  Analyzing  the  warfighter  requirements  in  a  QAW  framework  for 
performance  in  each  MUIRS  area  will  help  stakeholders  identify  software  quality 
attributes  that  need  to  be  communicated  to  potential  software  contractors  (Naegle, 
2006,  pp.  17-24). 

The  MUIRS  analysis  assists  the  QAW  process  by  focusing  on  those  elements 
that  are  too  often  typically  overlooked  during  the  requirements  generation  process. 
The  QAW  and  MUIRS  analysis  are  critical  to  the  software  design  process,  discussed 
in  the  next  section. 

Primary  Software  Acquisition  Problem  Area  Addressed 

MUIRS  primarily  addresses  the  immature  software  engineering  environment 
as  it  provides  an  analytic  approach  for  critical  sustainment  and  safety/security 
attributes  often  missing,  weakly  articulated,  or  vaguely  stated  in  the  requirements 
produced.  With  its  capabilities  and  performance  based  requirements  processes,  the 
DOD  significantly  depends  on  mature  engineering  environments  to  fill  the  gaps  left 
from  the  requirements  generation  and  communication  processes,  but  the  software 
engineering  environment  is  unable  to  do  so.  The  MUIRS  analysis  is  also  an  enabler 
for  the  QAW  and  ATAMsm  architectural  processes  discussed  next. 

Architectural  Tradeoff  Analysis  Methodology5"1 

The  SEI’s  ATAMsm  is  an  architectural  analysis  tool  designed  to  evaluate 
design  decisions  based  on  the  quality  attribute  requirements  of  the  system  being 
developed.  The  methodology  is  a  process  for  determining  whether  the  quality 
attributes  are  achievable  by  the  architecture  as  it  has  been  conceived  before 
enormous  resources  have  been  committed  to  that  design.  One  of  the  main  goals  is 
to  gain  insight  into  how  the  quality  attributes  trade-off  against  each  other  (Kazman, 
Kleim,  &  Clements,  2000,  p.  1). 

Within  the  SEP,  the  ATAM  provides  the  critical  Requirements  Loop  process, 
tracing  each  requirement  or  quality  attribute  to  corresponding  functions  reflected  in 
the  software  architectural  design.  Whether  ATAM  or  another  analysis  technique  is 
used,  this  critical  SEP  process  must  be  performed  to  ensure  that  functional-  or 
object-oriented  designs  meet  all  stated,  derived,  and  implied  warfighter 
requirements.  In  complex  systems  development  such  as  weapon  systems,  half  or 
more  than  half  of  the  total  software  development  effort  is  expended  in  the 
architectural  design  process.  Therefore,  the  DOD  PMs  must  ensure  that  the  design 
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is  addressing  requirements  in  context  and  that  the  resulting  architecture  has  a  high 
probability  of  producing  the  specified  warfighters’  capabilities  described  in  the  JCIDS 
documents. 

The  ATAM  focuses  on  quality  attribute  requirements,  so  it  is  critical  to  have 
precise  characterizations  for  each.  To  characterize  a  quality  attribute,  the  following 
questions  must  be  answered: 

•  What  are  the  stimuli  to  which  the  architecture  must  respond? 

•  What  is  the  measurable  or  observable  manifestation  of  the  quality 
attribute  by  which  its  achievement  is  judged? 

•  What  are  the  key  architectural  decisions  that  impact  achieving  the 
attribute  requirement?  (Kazman  et  al.,  2000,  p.  5) 

The  ATAM  is  designed  to  elicit  the  data  and  information  needed  to  adequately 
address  the  three  questions  above.  These  questions,  focused  on  requirements  and 
quality  attributes,  are  user-centric,  and  so  the  ATAM  scenarios  must  be  constructed 
by  the  user  community  (Naegle  &  Petross,  2007,  p.  25).  The  methodology  keys  on 
scenario  development  in  three  main  areas: 

•  Use  Case  Scenarios.  As  the  name  suggests,  these  scenarios 
describe  how  the  system  will  be  used  and  sustained  in  the  harshest 
environments  envisioned.  It  includes  all  interoperability  requirements 
and  duty  cycles  as  well. 

•  Growth  Scenarios.  Growth  scenarios  focus  on  known  and  anticipated 
system  change  requirements  over  the  intended  life  cycle.  These 
scenarios  include  upgrades  and  technology  refreshments  planned; 
interoperability  requirements,  such  as  inclusion  in  future  warfighting 
networks;  changes  in  sustainment  concepts,  and  other  system 
changes  expected  to  occur  over  time. 

•  Exploratory  Scenarios.  Exploratory  scenarios  focus  on  operations  in 
unusual  or  stressful  situations.  These  address  user  expectations  when 
the  system  is  degraded  or  operated  beyond  normal  limitations  due  to 
emergency  created  by  combat  environments.  These  scenarios  include 
Failure  Modes  and  Effects  Criticality  Analyses  (FMECA)  to  identify  the 
essential  functions  that  must  not  fail.  As  important  to  the  software 
engineers,  FMECA  also  identifies  those  enhancing  functions  that 
should  not  preclude  the  system  from  functioning  when  that  enhancing 
function  is  degraded  or  non-operational.  For  example,  the  Ml  Abrams 
tank  uses  the  ambient  temperature  as  an  enhancer  to  the  main  gun 
accuracy,  but  needs  the  ability  to  be  fully  operational  in  the  case  where 
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the  ambient  temperature  sensor  is  malfunctioning.  The  software 
engineers  need  that  information  to  properly  design  the  software. 

Test  cases  are  developed  out  of  the  scenarios,  which  firmly  link  the  test 
program  with  the  user  requirements  in  the  context  of  the  scenarios.  This 
methodology  also  helps  to  ensure  that  there  are  verification  events  for  software  and 
sustainment  requirements,  which  are  too  often  missing  from  the  testing  program. 

As  you  can  see  from  Figure  4,  the  ATAM  is  an  integrating  function  for  many  of 
the  tools  and  techniques  discussed  here.  It  is  designed  to  be  an  iterative  process 
and  would  be  most  effective  when  started  in  early  concept  development,  then 
continued  through  contract  award,  prototyping,  and  into  the  design  review  process. 


Capabilities-based  AT AM^1  Scenario  Development 
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Figure  4.  Quality  Attribution  Workshop  and  Architectural  Tradeoff  Analysis 
Methodology  Integration  into  Software  Life-Cycle  Management 

(Naegle  &  Petross,  2007,  p.  25) 

Primary  Software  Acquisition  Problem  Areas  Addressed 

The  ATAM  process  addresses  four  primary  problem  areas: 


•  The  scenario  development  provides  much  more  operational  context 
than  the  typical  OMS/MP  provides.  This  level  of  detail  helps  to 
compensate  for  the  immature  software  engineering  environment  and  is 
critical  for  the  proper  design  of  the  software  architecture. 
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•  The  ATAM  serves  as  a  very  effective  software  design  metric  function. 
With  the  software  development  team  using  50%  or  more  of  the 
available  resources  for  requirements  analysis  and  software  design 
before  the  Preliminary  Design  Review  (PDR),  it  is  critical  to  have  an 
effective  software  design  metrics  function.  Traditional  software  design 
metrics  focus  on  the  design  complexity  and  do  not  address  whether 
the  design  is  adequate  or  not.  ATAM  directly  links  the  user 
requirements  to  the  system  architectural  design. 

•  As  the  testing  program  is  developed  from  the  scenarios,  it  becomes 
difficult  to  omit  any  critical  testing  event.  In  addition,  the  software 
developer  understands  the  tests  or  verification  events  that  must  be 
passed  for  user  acceptance. 

•  By  integrating  the  MUIRS  analyses  into  the  ATAM  scenario 
development,  sustainability  and  safety/security  aspects  cannot  easily 
be  omitted  from  the  system  design.  As  the  testing  plan  flows  from  the 
scenarios,  the  MUIRS  design  elements  will  have  corresponding  test  or 
verification  events  identified  in  the  test  plan. 

Failure  Modes  and  Effects  Criticality  Analysis 

As  the  title  indicates,  this  analysis  methodology  is  designed  to  identify  system 
failure  modes  and  those  failures  effects  on  the  system,  and  ascertain  the  relative 
criticality  of  that  type  of  failure.  In  his  book  titled  Logistics  Engineering  and 
Management,  Benjamin  S.  Blanchard  (2004)  describes  FMECA  as  follows: 

Given  a  description,  both  in  functional  and  physical  terms,  the  designer 
needs  to  be  able  to  evaluate  a  system  relative  to  possible  failures,  the 
anticipated  modes  and  expected  frequency  of  failure,  their  causes, 
their  consequences  and  impact(s)  on  the  system  overall,  and  areas 
where  preventative  measures  can  be  initiated  to  preclude  such  failures 
in  the  future,  (p.  275) 

He  goes  on  to  state,  “The  FMECA  is  an  excellent  design  tool,  and  it  can  be 
applied  in  the  development  or  assessment  of  any  product  or  process”  (Blanchard, 
2004,  p.  276). 

Including  FMECA  scenarios  with  the  software  systems  and  subsystems 
provides  architectural  design  cues  to  software  engineers.  These  scenarios  provide 
analysis  for  designing  redundant  systems  for  mission-critical  elements,  “safe  mode” 
operations  for  survivability-  and  safety-related  systems,  and  drive  the  software 
engineer  to  conduct  “what  if”  analyses  with  a  superior  understanding  of  failure-mode 
scenarios.  For  example,  nearly  all  military  aircraft  are  “fly-by-wire,”  with  no  physical 
connection  between  the  pilot  controls  and  the  aircraft-control  surfaces,  so  basic 
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software  avionic  functions  must  be  provided  in  the  event  of  damage  or  power-loss 
situations  to  give  the  pilot  the  ability  to  perform  basic  flight  and  navigation  functions. 
Obviously,  this  would  be  a  major  design  driver  for  the  software  architect  (Naegle  & 
Petross,  2007). 

Primary  Software  Acquisition  Problem  Areas  Addressed 

The  primary  problem  areas  addressed  by  FMECA  include  requirements 
clarification  and  prioritization,  and  helping  to  ensure  a  sound  software  architecture 
design.  This  analysis  also  ensures  that  the  most  critical  software  systems  are 
designed  with  the  requisite  reliability  and  will  continue  to  function  in  degraded 
modes. 

As  previously  stated,  one  of  the  main  functions  of  performing  FMECA  is  to 
identify  those  software  functions  that  are  not  critical,  and  ensuring  that  failures  or 
anomalies  in  those  non-critical  functions  do  not  preclude  or  negatively  affect  system 
capabilities.  Today’s  systems  typically  have  numerous  enhancing  functions  that 
improve  performance  but  are  not  critical  and  the  software  developers  have  no  way  to 
discern  the  difference  between  a  critical  system  and  an  enhancing  one  without 
employing  FMECA. 

Integrating  the  Recommended  Tools,  Techniques,  and 
Processes  into  the  Defense  Acquisition  System 

The  tools,  techniques,  and  processes  were  specifically  selected  for  both  their 
ability  to  address  software-intensive  systems  development  problems,  and  their  ability 
to  integrate  with  the  DAS.  They  are  all  SEP  enablers  designed  to  improve  the  critical 
DAS  front-end  processes,  which  are  primarily  the  Government’s  responsibility. 

The  depiction  in  Figure  5  shows  the  processes  applied  at  the  latest  possible 
developmental  time  to  be  effective.  The  earlier  these  tools,  techniques,  and 
processes  occur,  the  more  effective  they  become.  This  representation  also  does  not 
show  the  iterative  cycles  of  QAW  and  ATAM  or  their  overlapping  nature. 
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QAW  &  ATAM  Integration  into  SW  Lifecycle  Management 

Mgt  QAW  ATAM  Post  Design  Mgt  Operations  & 

Support  Mgt 

Requirements  Design  Metrics  Development  Metrics 
Elicitation 


JCIDS 

Docs 

Explicit, 

Scenario 

Derived  & 

Development  & 

ICD 

Implied 

Prioritization 

CDD 

Requirements 

CPD 

Design  Reviews 

Activities 

RFP 

System  & 

Source 

Selection 


Prototype 

LUT& 

EUTE 


User  Testing  IOT&E 


Rapid 

Prototyping,  Accept,  Field 

Design  &  Code,  Build,  &  Support 

Analysis  Integrate,  Test 


Test  Case 
Development 

Tech 

Reviews/  SSR  SRR  SFR  PDR  CDR  DRR  FCA  PCA 

Audits 


Figure  5.  Quality  Attribution  Workshop  and  Architectural  Tradeoff  Analysis 
Methodology  Integration  into  Software  Lifecycle  Management 

(Naegle  &  Petross,  2007) 

As  depicted  in  Figure  5  the  QAW  and  ATAM  are  designed  to  address  critical 
requirements  and  design  front-end  processes,  where  the  Government  is  primarily 
responsible  for  the  process.  The  blue  arrow  shows  how  the  warfighters  and  user 
community  are  continuously  involved  throughout  the  process,  and  are  active 
participants  in  the  QAW  and  ATAM  processes.  This  is  distinctly  different  than  the 
traditional  DAS  where  there  is  little  formal  user  interaction  between  preparation  of 
the  JCIDS  documents  and  the  prototype  limited  user  tests  (LUT)/early  user  test  and 
evaluation  (EUT&E). 

The  user  communities  have  a  very  significant  role  in  driving  the  QAW  and 
ATAM  processes,  which  requires  more  user  resources  to  support  the  system 
development.  This  user  investment  in  the  DAS  is  becoming  more  critical  with  the 
development  of  more  software-intensive  and  complex  systems  of  all  kinds.  This 
investment  is  absolutely  necessary  to  avoid  Government  to  contractor 
misunderstanding  of  the  system  requirements  and  warfighter  expectations,  and 
would  significantly  reduce  the  cost  and  schedule  delays  associated  with  user 
dissatisfaction,  user-test  failure,  and  unnecessary  system  redesign. 
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Program  Management  Risk  Reduction 

These  tools,  techniques,  and  processes  will  not,  of  themselves,  produce  or 
guarantee  anything.  “An  architecture  analysis  method,  any  architecture  analysis 
method,  is  a  garbage-in-garbage-out  process.  The  ATAM  is  no  different.  It  crucially 
relies  on  the  active  and  willing  participation  of  the  stakeholders”  (Kazman  et  al., 

2000,  p.  63).  All  of  the  tools  and  techniques  described  and  recommended  in  this 
research  are  dependent  on  the  team  of  professional  stakeholders  conscientiously 
performing  their  critical  function  in  the  development  of  the  software-intensive  system. 

To  effectively  implement  the  recommended  tools,  techniques,  and  processes, 
the  program  management  team  must  be  professional,  disciplined  in  their  application 
of  the  SEP  and  skilled  in  integrating  the  tools  into  the  DAS.  In  a  word  used  by  the 
SEI,  the  team  must  be  mature.  The  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  mandates  certain  education  and  training  levels  for  the  professional 
workforce  performing  at  various  levels.  The  DOD  invests  significant  resources  in 
both  education  and  training  to  help  ensure  the  acquisition  workforce  competencies 
and  comply  with  the  DAWIA. 

The  DOD  also  evaluates  the  maturity  of  potential  software  developers  by 
requiring  an  evaluation  using  SEI’s  Capability  Maturity  Model-Integrated  (CMMI;  or 
equivalent)  for  most  software-intensive  system  acquisitions.  The  CMMI  is  a  five-level 
model,  and  the  software  developer  organization  under  evaluation  must  achieve  at 
least  a  level  three  by  an  independent  evaluation  team  to  be  eligible  to  be  awarded 
the  DOD  contract. 

As  mentioned  previously,  the  DOD  does  not  currently  require  the  PM  offices 
managing  software-intensive  systems  to  achieve  any  maturity  level  on  the  Software 
Acquisition  Capability  Maturity  Model  (SA-CMM).  The  team  effort  between  the 
Government  and  the  software  developer  strongly  suggests  that  both  the  PM  office 
and  the  software  developer  would  reduce  developmental  risk  by  demonstrating  an 
appropriate  level  of  maturity. 

Due  in  large  part  to  the  immature  software  engineering  environment,  each 
major  DOD  software  design  and  build  tends  to  be  unique.  That  means  that  the 
software  development  in  complex  systems  will  act  the  same  way  as  integrating  a 
new  technology  would,  and  the  resulting  program  risk  is  very  high.  The  software 
TRLs  have  little  meaning  in  this  type  of  environment,  so  risk  management  is  highly 
dependent  on  the  Government  and  software  development  teams’  abilities  to  manage 
the  system  software  development  as  a  new  technology  with  a  low  TRL. 
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A  significant  portion  of  the  risk  management  is  focused  on  the  Government 
and  software  development  teams.  As  the  software  TRLs  are  mostly  ineffective,  I 
would  recommend  the  further  development  of  software  Management  Readiness 
Levels  (MgtRLs)  to  mitigate  the  risks.  Part  of  the  management  risk  reduction  is 
already  in  place  with  the  DAWIA  requirements  and  the  software  developer  maturity 
levels  that  must  be  achieved.  Taking  this  further  by  including  the  PM  office  team 
maturity  and  integrating  the  tools,  techniques,  and  processes  recommended  in  this 
research,  I  have  outlined  the  basic  nine-tier  software  MgtRLs: 

•  Level  1 :  PM  team  and  software  developers  meet  all  professional 
certifications  and  adhere  to  DOD  policy  for  achieving  maturity  levels. 

•  Level  2:  PM  team  has  fully  developed  derived  and  implied 
requirements  using  a  QAW  approach. 

•  Level  3:  PM  team  has  conducted  a  self-evaluation  and  meets  the  SA- 
CMM  level  2  criteria. 

•  Level  4:  Post  contract  award,  the  PM  team  conducted  at  least  one 
iteration  of  ATAM. 

•  Level  5:  PM  team  conducts  pre  and  post  contract  award  ATAM 
iterations,  and  continues  ATAM  through  the  initial  program  design 
reviews. 

•  Level  6:  PM  team  has  achieved  level  2  in  an  externally  conducted  SA- 
CMM  evaluation.  Software  developer  has  achieved  level  3  (or  higher) 
in  an  externally  conducted  CMMI  evaluation. 

•  Level  7:  PM  team  has  achieved  level  3  in  an  externally  conducted  SA- 
CMM  evaluation.  Software  developer  has  achieved  level  3  (or  higher) 
in  an  externally  conducted  CMMI  evaluation. 

•  Level  8:  PM  team  has  achieved  level  4  in  an  externally  conducted  SA- 
CMM  evaluation.  Software  developer  has  achieved  level  4  (or  5)  in  an 
externally  conducted  CMMI  evaluation. 

•  Level  9:  PM  team  has  achieved  level  5  in  an  externally  conducted  SA- 
CMM  evaluation.  Software  developer  has  achieved  level  5  in  an 
externally  conducted  CMMI  evaluation. 
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As  with  the  TRLs,  I  would  recommend  that  the  target  MgtRL  for  any  PM  office 
managing  a  software-intensive  or  complex  system  would  be  level  6  to  help  reduce 
management  risk  to  an  acceptable  level.  This  is  significantly  more  overall 
management  maturity  than  what  DOD  prescribes  today,  but  this  research  strongly 
indicates  that  this  level  would  reduce  the  software-intensive  system  development 
risk. 

The  MgtRLs  suggested  focus  on  the  tools,  techniques,  and  processes 
researched,  but  there  is  likely  many  other  areas  that  could  be  researched  and  added 
into  the  nine-tier  model.  Areas  including  software  metrics,  quality  assurance, 
software-oriented  Integrated  Product  Teams  (IPTs),  contracting  plans,  and  others 
could  be  researched  and  included  in  the  appropriate  MgtRL  level. 
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